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ABSTRACT 

Developmental changes in the understanding and use of 
the logic of experimental control were addressed with three tasks in 
a longitudinal study. In all three tasks, understanding of 
experimental control was assessed by production measures (children 
were asked to test potential causal relations in a mul t ivariable 
situation) and by choice/evaluation measures (children were asketf to 
choose or evaluate tests made by hypothetical others). Tasks involved 
the rudder position on an airplane, the effect of the diameter of a 
spring on how far it stretches, and the determination of tree 
sickness. Children in Grades 2 to 5 produced experimental tests and 
chose/evaluated tests made by hypothetical others for those tasks. 
Cross-task and longitudinal analyses suggested that whereas the 
understanding of experimental control increased substantially between 
Grades 2 and 4, using this understanding to produce controlled tests 
lagged behind. On the basis of an analysis of some of the sources of 
individual differences, a training condition that facilitated the 
active representation and mental combination of variable dimensions 
was tested. Substantial test production improvement after training 
suggests that children's failure to produce controlled experiments 
arises from their failure to actively represent the problem space, 
rather than from a fundamental lack of understanding of the logic of 
experimental control. (Author/MDH) 
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PRODUCING SCIENTIFIC TESTS: WHAT DEVELOPS AND WHEN? ' 



Merry Bullock and Albert Ziegler 
Max Planck Institute for Psychological Research, 
Leopoldstrasse 24, 8000 Munchen 40, Germany 



Abstract 

Developmental changes in the understanding and use of the logic of experimental 
control were addressed with three tasks in a longitudinal study. Children in Grades 
2 to 5 produced experimental tests and chose/evaluated tests made by hypothetical 
others. Cress-task and longitudinal analyses suggested that whereas the 
understanding of experimental control increased substantially between Grades 2 
and 4, using this understanding to produce controlled tests lagged behind. On the 
basis of an analysis of some of the sources of individual differences, a training 
condition that facilitated the active representation ar.d mental combination of 
variable dimensions was tested. Substantial tesi production improvement afiei 
training suggests that children's failure to produce controlled experiments arises 
from their failure to actively represent the problem space, rather than from a 
fundamental lack of understanding of the logic of experimental control. 
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PRODUCING SCIENTIFIC TESTS: WHAT DEVELOPS AND WHEN? 

There is some controversy over when the ability to generate systematic and 
controlled tests to assess the influence of one or several potential causal factors 
becomes available. When given the task of constructing empirical tests in a 
multivariate context, preadolescents generally produce incomplete, uncontrolled 
tests (e.g., Inhelder & Piaget, 1958; Kuhn et al., 1988). Yet, younger grade 
schoolers seem to understand some of the basic concepts of experimentation. For 
example, they understand that one must vary the presence or absesee of a 
potential cause and can recognize when variables are not controlled (e.g., Bullock, 
1991), and they can produce a critical test to distinguish between two potential 
causes when no other variables are present (e.g., Sodian et al., 1991). 

There is also controversy over the reasons for children's failures to produce 
controlled tests: because they lack a conceptual understanding of experimental 
control, because they misconstrue the task to be one of hypothesis confirmation 
rather than hypothesis testing, or because they lack general analytic or 
representational skills. 

To address these issues, three tasks assessing the understanding of the logic 
of experimental control were presented in an ongoing longitudinal project on 
children's scientific reasoning. In all three tasks, understanding of experimental 
control was assessed by production measures (children were asked to test 
potential causa' relations in a multivariable situation) and by choice/evaluation 
measures (children were asked to choose or evaluate tests made by hypothetical 
others). To assess developmental changes, one task (story task) was presented in 



2 

each of three measurement years (covering grades 2 to 5); to assess cross-task 
consistency, two complementary tasks (Springs and Forest tasks) were presented 
in the second measurement year (grades 3-4). 

Task 1 : Story Task 

(presented in each measurement year with different contents) 

Children were asked to test the causal role of one of three variable 
dimensions. The number of test production and test choice responses that were 
controlled (focal dimension varied, other dimensions held constant) were assessed. 



Test Production example: 

Mr. Schmidt designs airplanes and wants to design a fuel-efficient airplane. He 
considers three things that might make a difference: 




What should Mr. Schmidt do to see if the rudder position makes a difference? 
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Test Choice example: 
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Which of these airplanes should Mr. Schmidt make to see if the 
rudder position makes a difference? 

Results 



Story Task: Percent controlled tests 
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Task 2: Springs Task 

(presented to 3rd and 4th graders) 

Children were asked to test the causal role of each of three variable dimensions ( 
(weight, spring diameter, spring material) and to evaluate tests made by others. 
The number of controlled tests produced and chosen over three trials were 
assessed. 



Test Production example : 




"Make a test to find out whether the diameter of a spring makes a difference in how 
far a spring stretches" 
Test Choice example : 

"Someone wanted to find out if spring diameter makes a difference and used 
springs 4 and 7, each time with the middle weight. Is this a good test? Why? " 
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Task 3: Forest Task 

(presented to 3rd and 4th graders) 

Children were asked to find out which of three chemicals made trees in a forest 
sick, and to evaluate tests made by others. The number of systematic and 
controlled tests produced (2 trials) and chosen (3 trials) was assessed. 



Test Production example : 




"Here are healthy and sick trees in a forest. You can test any tree for any of the 
three chemicals. Keep testing until you are sure you know which chemical is the 
cause." (Controlled strategy is to compare presence/absence of hypothesized 
cause in sick and healthy trees) 
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Test Choice example: 
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"Scientist A and B both concluded that beetles caused the trees to be sick. Who 

can be more sure of his/her conclusion and why?" 

Results 

Forest Task: Percent controlled tests 
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Summary over tasks 

Several findings are of special note. First, although children often failed to 
produce controlled, unconfounded tests, a sizeable proportion not only 
discriminated a confounded from an unconfounded test, but also explicitly justified 
this discrimination by explaining the need for variable control or control groups. 

Longitudinal analyses suggested that whereas an understanding of the 
importance of controlled tests increased substantially between the 2nd and 4th 
grades (8 and 10 years), the ability to apply this understanding when actually 
constructing tests lagged behind, and was not at ceiling even by the 5th grade (11 
years). Jhis suggests that although children possess the conceptual ability to 
construct controlled tests, they do not engage this understanding in their own 
production. 

This conclusion was supported by the results concerning consistency as 
shown in the following table: 



Proportion of children showing understanding of the logic of control across tasks 





No Tasks 


One Task 


Two Tasks 


All tasks 


Production 


51.5 


36.6 


11.9 


0 


Choice 


8.8 


20.6 


39.7 


30.9 


Justification 


35.1 


37.6 


21.6 


5.7 
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Why is test production performance worse and what can be done about it? 

Production, choice and justification responses across the three tasks were 
correlated with general skills (IQ, word span), scientific reasoning skills (scale from 
similar tasks presented earlier), and specific associated skills (logical reasoning, 
combinatory skills, avoiding premature closure). 



Correlations between scientific reasoning skills and other variables 





IQ 


SPAN 


LOGIC 


COMBI 


APC 


SCI 


Production 


.32 


.20 


.22 






.21 


Choice 


.41 


.32 




.22 


.21 


.25 


Justifications 


.47 


.41 




.23 


.24 


.41 



Each type of performance measure (production, choice, justification) was 
predicted by general ability and prior scientific reasoning skills. In addition, test 
understanding (assessed by choice/evaluation and justification tasks) was predicted 
by those skills that require active representation and manipulation of a problem 
space (memory span, combinatory thinking, avoiding premature closure). To see 
whether test production might be improved when a more salient representation of 
the problem space was available, a training manipulation was added to the story 
task presented to 4th and 5th graders. The purpose of the training manipulation 

was to facilitate the active representation of all the variables/dimensions and 

variable combinations relevant to test production in the subsequent story 

presentation. 

Training 

In the last measurement wave, the story task was presented twice, with 
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different contents. The training manipulation preceded the second administration of 
thQ^tory task and concerned the variables to be used in the second story task. In 
the training, children were introduced to three variable dimensions, each with two 
values. They (1) actively constructed all exemplars that could be made from 
combining each value of the three variable dimensions and (2) mentally generated 
exhaustive combinations of each specific variable with all other variables. 
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Story Task: Percent Controlled Tests 
Before and After Training 
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Conclusions 

Children acquire a conceptual understanding of the logic of experimental control 
during the grade school years, but cannot apply this understanding when actively 
producing experiments. One reason for this seems to be that they do not 
spontaneously actively generate an appropriate representatior, of the problem 
space. When induced to do so through a brief training, performance improves. 
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